Method and apparatus for failure resilient forwarding of data over a computer network

ABSTRACT

In one embodiment, the present invention is a method and an apparatus for failure-resilient forwarding of data over a computer network. In one embodiment, a marker is introduced into the data stream, e.g., at the sending node, and allows, in turn, forwarding nodes and/or receivers to efficiently track data stream reception. The marker functions as a checkpoint for the data transport process, and is identified and indexed at each forwarding node and receiver. Each receiver saves the marker prior to delivering data to an application, thereby designating a point in the data stream at which all preceding data is confirmed to have been delivered to the application. Thus, if a forwarding node fails, the receiver may request stream data from an alternate forwarding node by specifying to the alternate forwarding node to provide data starting from the marker.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.10/830,779, filed Apr. 23, 2004, (currently allowed) which is hereinincorporated by reference in its entirety.

BACKGROUND

The present invention relates generally to computer systems and computernetworks, and relates more particularly to content delivery overcomputer networks. Specifically, the present invention relates to amethod and apparatus for adaptive forwarding of data over a computernetwork.

FIG. 1 is a schematic illustration of one embodiment of a system 100 forforwarding data over a network. A wide range of end-to-end computingapplications (including overlay networks, end-system multicast, proxyservers, network address translation and protocol tunneling, amongothers) use intermediaries, or forwarding nodes 106 ₁-106 _(n) (e.g.,computing devices or routers), to route a stream 112 of data from asender 102 (e.g., a server) to one or more receivers 104. Receivers 104may in turn deliver the data to one or more computing applications 108.

A typical problem with a system such as the system 100 is that a failureor disruption at any forwarding node disrupts the end-to-end chain,resulting in incomplete data delivery to the receiver(s). This isespecially troublesome for large networks, as the probability of nodefailure increases with the number of forwarding nodes implemented.Conventional solutions for addressing node failure in a forwardingnetwork include source-based repair such as a Transmission ControlProtocol/Internet Protocol (TCP/IP) session between the data source andthe receiver, packet number-based retransmission requests, and variousapplication- and content-specific resiliency schemes (e.g., resumingFile Transport Protocol at a specific byte offset from the start of afile, or resuming a video transmission at a specific frame number).However, these conventional solutions are subject to a number oflimitations, including scalability limitations and the inability toadapt for use over a network using heterogeneous transports ordelivering generic (non-content-specific) data streams. Accordingly,they are not appropriate for delivering data over an adaptively changingnetwork using multiple point-to-point protocols in a content-independentmanner.

Thus, there is a need for a method and apparatus for failure-resilientforwarding of data over a computer network.

SUMMARY OF THE INVENTION

In one embodiment, the present invention is a method and an apparatusfor failure-resilient forwarding of data over a computer network. In oneembodiment, a marker is introduced into the data stream, e.g., at thesending node and, in turn, allows forwarding nodes and/or receivers toefficiently track data stream reception. The marker functions as acheckpoint for the data transport process, and is identified and indexedat each forwarding node and receiver. Each receiver saves the markerprior to delivering data to an application, thereby designating a pointin the data stream at which all preceding data is confirmed to have beendelivered to the application. Thus, if a forwarding node fails, thereceiver may request stream data from an alternate forwarding node byspecifying to the alternate forwarding node to provide data startingfrom the marker.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited embodiments of theinvention are attained and can be understood in detail, a moreparticular description of the invention, briefly summarized above, maybe had by reference to the embodiments thereof which are illustrated inthe appended drawings. It is to be noted, however, that the appendeddrawings illustrate only typical embodiments of this invention and aretherefore not to be considered limiting of its scope, for the inventionmay admit to other equally effective embodiments.

FIG. 1 is a schematic illustration of one embodiment of an end-to-endcomputing network;

FIG. 2 is a flow diagram illustrating one embodiment of a method forenabling failure-resilient forwarding of data from a sender to one ormore receivers according to the present invention;

FIG. 3 is a table illustrating one method of distributing content usingthe system illustrated in FIG. 2;

FIG. 4 is a flow diagram illustrating one embodiment of a method forrecovering lost data in a data stream; and

FIG. 5 is a high level block diagram of the present failure-resilientforwarding system that is implemented using a general purpose computingdevice.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe figures.

DETAILED DESCRIPTION

The present invention is a method and an apparatus for failure-resilientforwarding of data over a computer network. In one embodiment, a triggercondition, such as forwarding node failure, performance degradation,query, resource use imbalance and the like, initiates a networkadaptation to correctly resume transmission reception of a data stream.

FIG. 2 is a flow diagram illustrating the flow of data through oneembodiment of a method 200 for enabling failure-resilient forwarding ofdata from a sender to one or more receivers according to the presentinvention. The method 200 is initialized at step 202 and proceeds tostep 204, where a sending node or a forwarding node (e.g., sender 102 orany one of the forwarding nodes 106 of FIG. 1) obtains (in the case ofthe sender) or receives (in the case of the forwarding node) at least aportion of a data stream. In one embodiment, the data stream may simplybe a portion or an arbitrarily sized data segment of a much larger datastream. Namely, a sequence of “chunks” or “portions” of the larger datastream is being sent from the sending node to a receiving node. In step206, the sender or a forwarding node injects a marker into the portionof the data stream, and forwards the “marked” data stream, with orwithout further modification, to one or more next recipient nodes in thenetwork (e.g., one or more subsequent forwarding nodes or receivingnodes) via a point-to-point reliable transport protocol (e.g., aprotocol that is packet loss resilient, such as TCP/IP and the like).The marker designates a reference point in the generic data stream andin one embodiment is a recognizable bit field with a unique identifier.The marker may be recognized by reserved bit sequences, fixedinter-marker offsets, or an offset specified by a prior marker. Thus,markers may be periodically injected into the data stream, so that aplurality of marked data streams is transported through the network.

After injecting the marker in step 206, the method 200 branches off intoat least one of two possible subsequent processes. In steps 208-210, themethod executes steps in accordance with the function of a forwardingnode; in steps 209-214, the method 200 executes steps in accordance withthe function of a receiving node.

In step 208, the method 200 inquires if the recipient of the marked datastream is a forwarding node. If the method 200 determines that therecipient is forwarding node, the method 200 proceeds to step 210, wherethe method 200 inspects the received data stream, stores the data in alocal buffer of the forwarding node, and creates or updates a markerindex at the forwarding node. In one embodiment, the marker index thatthe method 200 updates comprises two key components: (1) a record of themost recently received marker; and (2) a record of each markerpreviously received and stored by the forwarding node. Once the method200 has updated the marker index, the method 200 forwards the markeddata stream to the next recipient(s) (e.g., one or more other forwardingnodes or receivers) in the network. The marked data stream is processedby the next recipient node(s) starting at the point in the method 200just following step 206, as indicated by the loop from step 210. Thus,all forwarding nodes receive the marked data stream, relay the markeddata stream to the next forwarding nodes or receivers, and index themarkers.

FIG. 3 is a schematic illustration of one embodiment of a marker index300 according to the present invention, such as the marker index updatedby the method 200 in step 210 of FIG. 2. In one embodiment, the markerindex 300 is a table. As illustrated, the marker index 300 stores, foreach marker (e.g., markers M₁-M₃), the marker's unique identifier andits position in the local buffer. As will be further described belowwith reference to FIG. 4, this stored information may be used to recoverdata lost, for example, due to a forwarding node failure.

Referring back to FIG. 2, if the method 200 concludes at step 208 thatthe recipient of the marked data stream is not a forwarding node, themethod 200 terminates.

Also after injecting the marker in step 206, the method 200 inquires instep 209 if the recipient is a receiving node. If the method 200concludes that the recipient is a receiving node, the method 200proceeds to step 212 and queues the stream data received by the receiveruntil the marker is encountered. In step 214, the method 200 saves themarker and delivers the queued data (i.e., all undelivered, non-markeddata preceding the marker in the data stream) to a process desiring theoriginal data stream (e.g., an application or a storage process).Alternatively, if the method 200 concludes in step 208 that therecipient is not a receiving node, the method 200 terminates.

In one embodiment, one or more nodes are both forwarding and receivingnodes. That is, a node may be adapted to both receive data for deliveryto an application, and also to forward the received data on to anothernode. Thus, the node is capable of executing both the forwarding and thereceiving methods contained within the method 200. Thus, although theforwarding and receiving processes (e.g., steps 208-210 and 209-214,respectively) are designated by sequential reference numerals, thereference numerals do not connote an order in which the processes occur.Therefore, those skilled in the art will appreciate that the forwardingand receiving methods are executed independently, and that the methodsmay actually occur simultaneously, or may occur one after the other inany order. Thus, the sequence of the reference numerals as they apply tosteps 208-216 is not intended to be limiting in any sense.

Thus, the markers injected into the data stream represent checkpointsfor the data transport process. By saving the markers at the receivers,the method 200 designates points in the data stream where all precedingdata has been delivered, reliably and in order, to the waitingapplication. The method 200 also serves the function of designatingpoints in the data stream where any succeeding data has yet to bedelivered. This saved marker information may be used to recover datalost, for example, due to a forwarding node failure.

FIG. 4 is a flow diagram illustrating one embodiment of a method 400 forrecovering lost data in a data stream. For example, the method 400 maybe executed in the event that a forwarding node (e.g., a forwarding node106 of FIG. 1) fails (e.g., due to disconnection from the network orpower failure) and thus ceases to forward data to subsequent recipients.The method 400 is initialized at step 402 and proceeds to step 404,where the method 400 identifies a forwarding node failure and connects areceiver (or subsequent forwarding node) to an alternate forwardingnode, or a “backup node” (e.g., a node that preceded the failed node inthe routing path). Alternatively, the method 400 may connect thereceiver to any “sister” node of the failed forwarding node that isstill receiving the data stream. In one embodiment, the backup node isselected for efficiency. For example, if the failed node is node X_(n)in FIG. 1, then the backup node can be selected to be node X₁ or nodeX_(n+1). The selection of the proper node can be based on distance,delay, computational cost and the like.

The method 400 then proceeds to step 406, where the method 400 requests,from the backup node, the stream data starting from the last marker, M,saved by the receiver. In an alternative embodiment, the method 400 mayrequest the stream data starting from a specified position after thelast marker M (e.g., three bits after the marker M). The requestincludes the unique identifier for the marker M. In step 407, the method400 inquires if the backup node will accept the request presented instep 406. If the backup node rejects the request, the method 400 returnsto step 404 and connects to another backup node. Alternatively, if thebackup node accepts the request in step 407, the method 400 enables thebackup node to look up the marker M in the backup node's marker index.If the marker M is present, the backup node begins sending the markeddata stream, using the location of the marker M in its local buffer asthe starting point. In one embodiment, any data residing in the localbuffer past the point of the marker M is discarded.

In step 408, the method 400 resets a queue “write pointer” for thereceiver to a position immediately following the marker M. The method400 also erases data following the write pointer in the local buffer,and the receiver will now start queuing data over the new connectionfrom the new forwarding node. As the marked data stream arrives at thereceiver over the new connection from the backup node, the arriving datastream overwrites any data following the marker M in the receiver'slocal buffer. In an alternative embodiment, the method 400 may requestdiscrete portions of the marked data stream from multiple backup nodes.

At step 410, the method 400 inquires if the next marker, M+1, hasarrived at the receiver. If the next marker M+1 has arrived, the method400 delivers data queued by the receiver (minus the marker M) to anapplication requesting the data at step 412. If the next marker M+l hasnot arrived, the method 400 continues to queue data over the newconnection from the backup node. Those skilled in the art will recognizethat steps 408-412 of the method 400 are steps typically executed by areceiver node; they have been discussed here, in the context of themethod 400, to illustrate the method by which the receiver mode mayimplement such steps in conjunction with the recovery of lost data.

The method 400 is therefore able to repair failures accurately andefficiently by resuming data transmission at the point of interruption.Moreover, as the repair only requires communication with a nearbyforwarding/backup node, repair paths are short and network load isfairly distributed. The method 400 also works at the application layerwith any reliable point-to-point transport protocol, can leverageexisting point-to-point protocols, and may allow reframing andmulti-protocol forwarding. Thus, the method 400 works independently oftransport protocols, as well as independently of data stream content.

FIG. 5 is a high level block diagram of the present failure-resilientforwarding system that is implemented using a general purpose computingdevice 500. In one embodiment, a general purpose computing device 500comprises a processor 502, a memory 504, a failure-resilient forwardingmechanism or module 505 and various input/output (I/O) devices 506 suchas a display, a keyboard, a mouse, a modem, and the like. In oneembodiment, at least one I/O device is a storage device (e.g., a diskdrive, an optical disk drive, a floppy disk drive). It should beunderstood that the failure-resilient forwarding mechanism 505 can beimplemented as a physical device or subsystem that is coupled to aprocessor through a communication channel.

Alternatively, the failure-resilient forwarding mechanism 505 can berepresented by one or more software applications (or even a combinationof software and hardware, e.g., using Application Specific IntegratedCircuits (ASIC)), where the software is loaded from a storage medium(e.g., I/O devices 506) and operated by the processor 502 in the memory504 of the general purpose computing device 500. Thus, in oneembodiment, the failure-resilient forwarding mechanism 505 and theassociated methods described herein with reference to the precedingFigures can be stored on a computer readable medium or carrier (e.g.,RAM, magnetic or optical drive or diskette, and the like).

Although the methods described herein have been discussed with referenceto system recovery from node failures, those skilled in the art willappreciate that the present invention may have other applications in thefield of content delivery. For example, the present invention may beimplemented to assure reliable data delivery with any networkreconfiguration, and for any reason. Other reconfiguration techniquesmay include finding a backup node using a centralized or distributedregistry of nodes (e.g., a known server or a Domain Name Service (DNS)lookup), a distributed hash table lookup, or a broadcast search, amongothers. Other reasons for network reconfiguration may include respondingto performance degradation, optimization of network resource utilizationand load balancing, among others.

Thus, the present invention represents a significant advancement in thefield of content delivery. A method and apparatus are provided thatenable efficient, failure-resilient forwarding of data over a network.The network is able to accurately and efficiently resume datatransmission at the point of interruption, without transmittingredundant or out-of-order data to a receiver. To an applicationrequesting data from a sender, the failure and recovery of the systemare substantially transparent. Moreover, the methods of the presentinvention are not application specific, but may be adapted for use withany type of data stream, regardless of content, and with any type ofreliable transport protocol.

While foregoing is directed to the preferred embodiment of the presentinvention, other and further embodiments of the invention may be devisedwithout departing from the basic scope thereof, and the scope thereof isdetermined by the claims that follow.

1. A method for resuming interrupted transport of data over a networkfrom a sender node to a receiver node, the method comprising: connectingto an intermediate node in the network; and requesting data from theintermediate node starting from a last marker saved by the receivernode.
 2. The method of claim 1, further comprising: resetting a receiverqueue write pointer to a position immediately following the last marker;and queuing data from the intermediate node to the receiver node.
 3. Themethod of claim 1, further comprising: delivering data queued by thereceiver node when a next marker is received from the intermediate node,wherein the data that is queued is delivered to an application.
 4. Themethod of claim 1, wherein said requesting data from the intermediatenode comprises: specifying a unique identifier for the last marker; andlooking up the last marker in a local buffer of the intermediate node.5. The method of claim 4, further comprising: discarding any data in thelocal buffer received after the last marker.
 6. The method of claim 1,further comprising: overwriting data following the last marker in alocal buffer of the receiver node, wherein the data is overwritten withthe data being received from the intermediate node.
 7. A computerreadable medium containing an executable program for resuminginterrupted transport of data over a network from a sender node to areceiver node, where the program performs: connecting to an intermediatenode in the network; and requesting data from the intermediate nodestarting from a last marker saved by the receiver node.
 8. The computerreadable medium of claim 7, further comprising: resetting a receiverqueue write pointer to a position immediately following the last marker;and queuing data from the intermediate node to the receiver node.
 9. Thecomputer readable medium of claim 7, further comprising: delivering dataqueued by the receiver node when a next marker is received from theintermediate node, wherein the data that is queued is delivered to anapplication.
 10. The computer readable medium of claim 7, wherein saidrequesting data from the intermediate node comprises: specifying aunique identifier for the last marker; and looking up the last marker ina local buffer of the intermediate node.
 11. The computer readablemedium of claim 10, further comprising: discarding any data in the localbuffer received after the last marker.
 12. The computer readable mediumof claim 7, further comprising: overwriting data following the lastmarker in a local buffer of the receiver node, wherein the data isoverwritten with the data being received from the intermediate node. 13.A system for resuming interrupted transport of data over a network froma sender node to a receiver node, comprising: means for connecting to anintermediate node in the network; and means for requesting data from theintermediate node starting from a last marker saved by the receivernode.
 14. The system of claim 13, further comprising: means forresetting a receiver queue write pointer to a position immediatelyfollowing the last marker; and means for queuing data from theintermediate node to the receiver node.
 15. The system of claim 13,further comprising: means for delivering data queued by the receivernode when a next marker is received from the intermediate node, whereinthe data that is queued is delivered to an application.
 16. The systemof claim 13, wherein said means for requesting data from theintermediate node, specifies a unique identifier for the last marker,and looks up the last marker in a local buffer of the intermediate node.17. The system of claim 16, further comprising: means for discarding anydata in the local buffer received after the last marker.
 18. The systemof claim 13, further comprising: means for overwriting data followingthe last marker in a local buffer of the receiver node, wherein the datais overwritten with the data being received from the intermediate node.